Stereoscopic Camera Apparatus

ABSTRACT

In order to provide a stereoscopic camera apparatus capable of rapidly and accurately detecting a three-dimensional object even when the three-dimensional object abruptly appears in an angle of view, a stereoscopic camera apparatus 100 includes: three-dimensional object detecting means 105, 106, 107; to-be-covered object detecting means 108 for detecting a to-be-covered object having a known shape; and to-be-covered object covering-detecting means 109 and 110 for detecting whether or not the three-dimensional object detected by the three-dimensional object detecting means covers the to-be-covered object when the to-be-covered object is detected by the to-be-covered object detecting means 108. When it is detected that the three-dimensional object covers the to-be-covered object, the to-be-covered object covering-detecting means 109 and 110 detect the three-dimensional object which covers the to-be-covered object by performing a process for determining whether or not the three-dimensional object is present in a simpler way, as compared to a case where the to-be-covered object covering-detecting means 109 and 110 detect that the three-dimensional object does not cover the to-be-covered object.

TECHNICAL FIELD

The present invention relates to a stereoscopic camera apparatus whichdetects a target using images captured from plural image sensors.

BACKGROUND ART

As a method of detecting a target using images captured from pluralimage sensors, a method of calculating a distance of each pixel of animage in a real space using a pair of images according to the principleof triangulation so as to detect a three-dimensional object is known inthe related art (PTL 1). In this method, groups of pixels whosepositions on the images are close to each other and whose calculateddistances in a real space are also close to each other are detected, anda group of pixels having a predetermined size or more is detected as athree-dimensional object. As a result, a three-dimensional object can bedetected with high accuracy.

In addition, as another method of accurately detecting a target usingimages captured from plural image sensors, a method of detecting anobject moving at a given speed on a pedestrian crossing and identifyingthe object with high accuracy as a pedestrian is known in the relatedart (PTL 2). In this method, a pedestrian moving on a pedestriancrossing is detected with high reliability by increasing the detectionsensibility thereof such that collision with the pedestrian. is avoided.As a result, a collision accident with a pedestrian on a pedestriancrossing where an accident is likely to occur can be prevented.

CITATION LIST Patent Literature

-   PTL 1: JP-A-5-265547-   PTL 2: JP-T-2011-525005

SUMMARY OF INVENTION Technical Problem

However, in the techniques disclosed in PTLS 1 and 2, for example, in acase where a pedestrian is detected using a vehicle-mounted stereoscopiccamera apparatus to avoid collision with the pedestrian, there is aproblem in that, when a vehicle turns right or left such that apedestrian abruptly appears in an angle of view, the pedestrian cannotbe rapidly identified as a target.

That is, PTL 1 discloses a technique of detecting a group of positionsclose to each other by a distance shorter than the distance of eachpixel as a three-dimensional object. However, in practice, during thedetection of a three-dimensional object, only when a candidate for thethree-dimensional object is continuously or intermittently detected in aregion near a real space a predetermined number of times within apredetermined amount of time, this candidate can be identified as acontrol target. This method is to prevent the erroneous detection of athree-dimensional object and to prevent erroneous braking or the likebased on the erroneously detected three-dimensional object even when awrong distance is calculated due to mismatching during the calculationof the distance of each pixel. This method is generally used. In thisway, in the method disclosed in PTL 1, a relatively long period of timeis required to detect a three-dimensional object. Therefore, forexample, when a vehicle turns right or left such that a pedestrianabruptly appears in an angle of view, this pedestrian cannot be rapidlyidentified as a control target.

In addition, PTL 2 discloses a technique of detecting a pedestriancrossing so as to detect a three-dimensional object moving at a givenspeed or higher on the pedestrian crossing as a pedestrian with highreliability. However, as in the technique disclosed in PTL 1, it isconsidered that, in order to prevent the erroneous detection of athree-dimensional object, a given period of time is required until athree-dimensional object is identified as a control target after thedetection thereof. In addition, when means for detecting athree-dimensional object is an image sensor, a relative speed cannot bedirectly calculated. Therefore, a given period of time is also requiredto determine whether or not the three-dimensional object moves at agiven speed or higher. Further, even when a three-dimensional object isdetected using a millimeter-wave radar capable of directly calculating arelative speed, a moving speed in a horizontal direction cannot bedirectly calculated. Therefore, a long period of time is required forthe determination of a control target.

Accordingly, in the techniques disclosed in PTLs 1 and 2, when a vehicleturns right or left such that a pedestrian abruptly appears in an angleof view, a warning to a driver may be delayed, or the operation ofautomatic braking may be delayed.

The present invention has been made in consideration of theabove-described circumstances, and an object thereof is to provide astereoscopic camera apparatus capable of rapidly and precisely detectinga three-dimensional object even when the three-dimensional objectabruptly appears in an angle of view.

Solution to Problem

In order to solve the problem, the present invention provides astereoscopic camera apparatus including: three-dimensional objectdetecting means for calculating at least a distance to athree-dimensional object in a real space, a horizontal position, and awidth by stereoscopy based on images captured from first and secondimage sensors; to-be-covered object detecting means for detecting ato-be-covered object having a known shape based an image captured fromone of the first and second image sensors; and to-be-covered objectcovering-detecting means for detecting whether or not thethree-dimensional object detected by the three-dimensional objectdetecting means covers the to-be-covered object when the to-be-coveredobject is detected by the to-be-covered object detecting means, in whichwhen it is detected that the three-dimensional object covers theto-be-covered object, the to-be-covered object covering-detecting meansdetects the three-dimensional object which covers the to-be-coveredobject by performing a process for determining whether or not thethree-dimensional object is present in a simpler way, as compared to acase where it is detected that the three-dimensional object does notcover the to-be-covered object.

Advantageous Effects of Invention

According to the present invention, not only the detection of athree-dimensional object using stereoscopy but also the determination ofwhether or not a three-dimensional object covers a to-be-covered objecthaving a known shape are performed. When it is detected that thethree-dimensional object covers the to-be-covered object, a processusing the to-be-covered object covering-detecting means for determiningwhether or not the three-dimensional object is present is performed in asimpler way. Therefore, the presence of the three-dimensional object ata position of the to-be-covered object can be determined within a shortperiod of time. Accordingly, when a vehicle turns right, or left suchthat a pedestrian abruptly appears in an angle of view, a warning to adriver or the operation of automatic braking can be accuratelyperformed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a stereoscopic camera apparatusaccording to Example 1.

FIG. 2 is a diagram showing a principle of measuring a distance in thestereoscopic camera apparatus.

FIG. 3 is a diagram showing a distance calculating process which isperformed by a distance calculating unit included in the stereoscopiccamera apparatus according to Example 1.

FIG. 4 is a matching decree graph of each pixel which is obtained by thedistance calculating unit included in the stereoscopic camera apparatusaccording to Embodiment 1.

FIG. 5 is a diagram showing a method of setting pedestrian crossingdetection frames which is performed by a pedestrian crossing detectingunit included in the stereoscopic camera apparatus according to Example1.

FIG. 6 is a diagram showing a method of detecting a pedestrian crossingwhich is performed by the pedestrian crossing detecting unit included inthe stereoscopic camera apparatus according to Example 1 and showing aluminance projection graph and a luminance distribution graph which areobtained using the method.

FIG. 7 is a flowchart showing the procedure of a control targetdetermining unit included in stereoscopic camera apparatus according toExample 1.

FIG. 8 is a block diagram showing a stereoscopic camera apparatusaccording to Example 2.

FIG. 9 is a diagram showing a method of setting pedestrian crossingdetection frames which is performed by a pedestrian crossing detectingunit included in the stereoscopic camera apparatus according to Example2.

FIG. 10 is a flowchart showing the procedure of a control targetdetermining unit included in the stereoscopic camera apparatus accordingto Example 2.

DESCRIPTION OF EMBODIMENTS

Hereinafter, each example of an embodiment of a stereoscopic cameraapparatus according to the present invention will be described withreference to the drawings. In the following description, a stereoscopiccamera apparatus including two cameras will be described as an example,but the scope of the present, invention is not limited thereto. Forexample, the present invention can be applied to a stereoscopic cameraapparatus including three or more cameras.

EXAMPLE 1

FIG. 1 shows the overall configuration of a stereoscopic cameraapparatus 100 according to Example 1. The stereoscopic camera apparatus100 according to Example 1 detects a pedestrian as a three-dimensionalobject, detects a pedestrian crossing as a to-be-covered object,determines and selects a control target. That is, as clearly seen fromFIG. 1, the stereoscopic camera apparatus 100 according to Example 1includes two imaging units of a camera 101 and a camera 102 and acquiresimages captured from the camera 101 and the camera 102 through an imageacquiring unit 103 and an image acquiring unit 104. In the distancecalculating unit 105, a distance in a real space is calculated for eachpreset fixed block size using the images acquired by the image acquiringunit 103 and the image acquiring unit 104, respectively. In athree-dimensional object extracting unit 106, the pedestrian as thethree-dimensional object is extracted using the distance obtained by thedistance calculating unit 105. In a three-dimensional object trackingunit 107, three-dimensional object information obtained by thethree-dimensional object extracting unit 106 is tracked in achronological order, a relative speed is calculated, and the results offiltering various parameters of the three-dimensional object are output.In a pedestrian crossing detecting unit 108, using the filteredthree-dimensional object information obtained by the three-dimensionalobject extracting unit 107 and the image obtained by the image acquiringunit 104, the pedestrian crossing near the three-dimensional object isdetected, and whether or not the pedestrian as the three-dimensionalobject covers the pedestrian crossing as the to-be-covered object isdetermined. In a control target determining unit 109, using theinformation regarding whether or not the pedestrian crossing is presentnear the three-dimensional object obtained by the pedestrian crossingdetecting unit 108, whether or not a tracked three-dimensional objectobtained by the three-dimensional object tracking unit 107 is acandidate for a control target is determined. In a control targetselecting unit 110 a control target is selected among control targetcandidates obtained by the control target determining unit 109, andinformation of the control target as a selected target is transmitted toa vehicle control device 111.

Hereinafter, the configuration of each unit shown in FIG. 1 will bedescribed in detail.

The camera 101 and the camera 102 are installed near a rear-view mirrorinstalled inside a vehicle so as to be distant from each other in ahorizontal direction by a given distance, to be positioned at the sameheight position, and to face the front of the vehicle. These cameras 101and 102 include an image sensor such as COD or CMOS and are synchronizedwith each other and adjusted so as to be sampled at the same timing.

The image acquiring unit 103 acquires the image captured from the camera101 and converts a luminance value to a digital value such that imageprocessing can be performed in the subsequent processing unit. The imageacquiring unit 104 acquires the image captured from the camera 102 andconverts a luminance value to digital data such that image processingcan be performed in the subsequent processing unit. In addition, theimage acquiring unit 103 and the image acquiring unit 104 correct theimage captured from the camera 101 and the image captured from thecamera 102 so as to remove differences in imaging environment andimaging characteristics between the two cameras, and transmit the imagedata to the subsequent process.

The distance calculating unit 105 divides the image acquired by theimage acquiring unit 104 into blocks having a preset fixed block size(for example, 4×4 [pix]) and calculates a distance in a real space foreach divided block. FIG. 2 shows a general principle of measuring adistance using the stereoscopic camera apparatus. In the drawing,reference numeral D represents a distance from an installation positionof a lens 302 and a lens 303 to a measurement point 301; referencenumeral f represents a distance (focal length of the lens 302) betweenthe lens 302 and an imaging plane 304 and a distance (focal length ofthe lens 303) between the lens 303 and an imaging plane 305; referencenumeral b represents a distance (base-line length) between center of thelens 302 and the center of the lens 303; and reference numeral drepresents a difference (disparity) between a position where themeasurement point 301 is imaged on the imaging plane 304 through thelens 302 and a position where the measurement point 301 is imaged on theimaging plane 305 through the lens 303. The following equation isestablished among the above symbols according to a relationship oftriangle similarity.

D=b×f/d  (1)

In the equation (1), the focal length f and the base-line length b areconstants determined based on the configuration of the cameras.Therefore, in order to obtain the distance D, only the disparity d whichis a difference in vision between the left and right lenses needs to beobtained. A method of obtaining this disparity d will be described usingFIG. 3. In Example 1, the image captured from the camera 101 installedon the left side will be referred to as “left image”, and the imagecaptured from the camera 102 installed on the right side will bereferred to as “right image”. Further, the left image is set as “baseimage”, and the right image is set as “reference image”.

An image 201 which is obtained by dividing the base image into a fixedblock size (for example, a size of 4×4 [pix]) is compared to an image202 which has the same height and the same size as those of the image201 on the reference image so as to calculate a matching degree betweenthe images. The matching degree between the images can be calculatedusing, for example, the sum of the absolute differences (SAD) betweenthe luminances of the respective pixels. The calculation of the matchingdegree is repeated while shifting the pixels one by one in a range fromthe image 202 to the image 203, which is distant from the image 202 by apredetermined number of pixels, to search for a block having the highestmatching degree. That is, when The calculation of the matching degree isrepeated while shifting the pixels one by one in a range from the image202 to The image 203, a graph shown in FIG. 4 is obtained, and themaximum value 301 is a disparity having the highest matching degree. Asa result, since the disparity d can be obtained from the differencebetween the images, the distance D of each block in a real space can becalculated using the equation (1). The peak of the SAD results aroundthe disparity having the highest matching degree is obtained using, forexample, quadratic function approximation. As a result, the disparitycan be calculated, instead of in units of pixels, in units ofsub-pixels.

The three-dimensional object extracting unit 106 extracts thethree-dimensional object based on a depth map which is calculated byobtaining a distance for each block of the entire region of the imagesusing the distance calculating unit 105. First, distance histograms arecreated for each column of blocks, and when the number of distancehistograms having a peak is a threshold value or more, it can beconsidered that a three-dimensional object candidate is present in thecolumn. Next, in a column adjacent to the above column, athree-dimensional object candidate is present, and when a distance of ahistogram having a peak is a threshold value or less, this column isgrouped as the same three-dimensional object. Finally, when the width ofthe grouped three-dimensional object is a threshold value or more, therecolumns are registered as a three-dimensional object. After the group isregistered as a three-dimensional object, blocks around the distances ofthe columns which form the same three-dimensional object are grouped onthe depth map, and a distance to the three-dimensional object in a realspace is calculated based on positions on the depth map (a lower end, anupper end, a left end, and a right end on the image) and the average ofthe grouped distances. Hereinafter, the three-dimensional objectobtained as described above will be referred to as an extractedthree-dimensional object having a position, a distance, and the like asparameters. The peak value of the histograms varies depending on thedistance to the three-dimensional object. Therefore, in order toaccurately extract various three-dimensional objects having differentdistances, the above-described threshold values are appropriatelychanged depending on the distances.

The three-dimensional object tracking unit 107 tracks the extractedthree-dimension object obtained by the three-dimensional objectextracting unit 106 in a chronological order, calculates a relativespeed based on the change amount of distance, filters the parameterssuch as a distance and a relative speed, and outputs a trackedthree-dimensional object having a position, a distance, and a relativespeed, which are filtered, as parameters. Specifically, the followingprocess is repeated. That is, when a tracked three-dimensional object ofprevious processing frame is present, this tracked three-dimensionalobject is compared to an extracted three-dimensional object detected inthe present processing frame. When a difference of each parameter is athreshold value or less, it is determined that the trackedthree-dimensional object matches the extracted three-dimensional object,and parameters of the tracked three-dimensional object of the previousprocessing frame are updated using parameters of the extractedthree-dimensional object of the present processing frame. The details ofan updating method will not described in detail, but can be realized,for example, by setting the extracted three-dimensional object as anobserved value using a Kalman filter and obtaining an error variancefrom actually measured values. In addition, the extractedthree-dimensional object which does not match any trackedthree-dimensional object of the previous processing frame is newlyregistered as an initially detected tracked three-dimensional object.

The pedestrian crossing detecting unit 108 determines whether or noteach tracked three-dimensional object covers the pedestrian crossingusing the tracked three-dimensional object, which is obtained by thethree-dimensional object tracking unit 107, and the image which isobtained by the image acquiring unit 103. First, as shown in FIG. 5,pedestrian crossing detection frames 501 and 502 for determining whetheror not the pedestrian crossing is present are set on left and rightsides of position information 500 which is obtained as a parameter ofthe tracked three-dimensional object. At this time, when an image is notpresent on left side or the right side of the three-dimensional objector when the sizes of pedestrian crossing detection frames to be set onthe left side or the right side of the three-dimensional object aresmaller than a preset threshold value, the pedestrian crossing detectionframes are not set. As positions of setting the pedestrian crossingdetection frames, upper ends thereof are positioned in the middle of thethree-dimensional object, lower ends thereof are positioned below lowerends of the three-dimensional object, and the width thereof isexperimentally determined.

Next, as shown in FIG. 6, the pedestrian crossing detection frames 501and 502 is divided into strips 601 (for example, widths: 3 [pix]).Regarding the divided strips, an average luminance (for example, averageof luminances of the widths 3 [pix]) is obtained for each row, and thisprocess is performed in an amount corresponding to the height of thepedestrian crossing detection frames. Using this average, the noise ofthe image can be reduced. A luminance projection graph 602 obtained asdescribed above is created for all the strips, and histograms of thevalues are created to create a luminance distribution graph 603. Byobtaining a variance of a luminance peak distribution obtained at thistime, whether or not the pedestrian crossing is present is determinedbased on the size of the variance. For example, in the case of thepedestrian crossing, contrast is concentrated on two portions includinga bright portion and a dark portion, and thus the variance thereofincreases. Conversely, in a region other than the pedestrian crossing, adistribution is shown in a portion other than a highly contrastingportion, and thus the variance thereof decreases. Using thischaracteristic, a threshold value for determining whether or not thepedestrian crossing is present is experimentally set and determined. Atthis time, when it is determined from one of the pedestrian crossingdetection frames that the pedestrian crossing is present, it isdetermined that the three-dimensional object covers the pedestriancrossing. In this way, by performing the determination on a region nearthe tracked three-dimensional object, a process load can be suppressedas compared to a case where whether or not the pedestrian crossing ispresent is determined using the entire region of the images.

The control target determining unit 109 determines a candidate for acontrol target using the tracked three-dimensional object obtained bythe three-dimensional object tracking unit 107 and pedestrian crossingcovering information for each tracked three-dimensional object obtainedby the pedestrian crossing detecting unit 108. FIG. 7 shows a processflow. First, when the pedestrian crossing covering information obtainedby the pedestrian crossing detecting unit 108 contains the content thatthe tracked three-dimensional object does not cover the pedestriancrossing, a normal reliability verifying process 702 is performed. Inaddition, when the pedestrian crossing covering information contains thecontent that the tracked three-dimensional object covers the pedestriancrossing, a simple reliability verifying process is performed.

In the normal reliability verifying process 702, whether or not theextracted three-dimensional object matches the tracked three-dimensionalobject through, for example, previous continuous 5 or more processingframes is verified such that only the tracked three-dimensional objectsatisfying a condition is registered as a control candidatethree-dimensional object. As a result, the risk of unintentionallycontrolling a detected three-dimensional object based on the influenceof a wrong distance, which is caused by artifacts due to noise ormismatching in the distance calculating unit 105, can be avoided. On theother hand, in the simple reliability verifying process 701, basically,the tracked three-dimensional object is registered as a controlcandidate three-dimensional object without verification. The reason isas follows. The information containing the content that the pedestriancrossing is covered and the circumstance that the three-dimensionalobject is detected accurately show the presence of the three-dimensionalobject. Therefore, the presence of the three-dimensional object can beaccurately shown without a long period of verification. In the simplereliability verifying process 701, it is sufficient that theregistration of the control candidate three-dimensional object isperformed more rapidly as compared to the normal reliability verifyingprocess 702, and the verifying process is not necessarily omitted. Forexample, when the accuracy of the pedestrian crossing coveringinformation is low, the registration of the control candidatethree-dimensional object can be performed under a condition where thepedestrian crossing covered state is continuously detected twice.

In the control target selecting unit 110, the most suitable controltarget is selected among the control candidate three-dimensional objectsdetermined, by the control target determining unit 109. Specifically, asteering angle, a yaw rate, and a vehicle speed are acquired through aCAN. In addition, a white line is identified from the images obtained byimage acquiring units. These pieces of information are combined toestimate a traveling line of a vehicle, and a control candidatethree-dimensional object on the traveling inc which is positioned on themost front side is output as a control target.

The vehicle control device 111 determines whether or not the risk ofcollision is present based on control target. information such as adistance in a real space or a relative speed which is transmitted fromthe stereoscopic camera apparatus 100 through a CAN bus or the like.When the risk is present, the vehicle control device 111 emits an alarmto urge a driver to take a collision avoiding action. When the collisioncannot be avoided in the end, automatic braking is operated to avoid thecollision or to reduce the impact.

As described above, in the stereoscopic camera apparatus according toExample 1, whether or not the three-dimensional object covers thepedestrian crossing is also added as the information. When it isdetermined that the three-dimensional object covers thethree-dimensional object, the registration of the control candidatethree-dimensional object is performed through the simple reliabilityverifying process. Therefore, the registration of the control candidatethree-dimensional object can be rapidly performed. When a vehicle turnsright or left such that a three-dimensional object abruptly appears inan angle of view of the cameras, an accident can be prevented inadvance. In addition, in the stereoscopic camera apparatus according toExample 1, the distance to a known shape can be calculated usingstereoscopy, for example, even when a traffic sign image is differentfrom a normal image due to contamination, scratching, or shadow.Therefore, whether or not the detection is performed due to the coveringof the three-dimensional object or simply due to contamination,scratching, or shadow can be determined, and erroneous detection can beprevented. Further, in the stereoscopic camera apparatus according toExample 1, the three-dimensional object extracting process is performedbefore the pedestrian crossing detecting process. Therefore, theextraction of the three-dimensional object can be accurately performed.

EXAMPLE 2

FIG. 8 shows the overall configuration of a stereoscopic cameraapparatus 800 according to Example 2. The stereoscopic camera apparatus800 includes two imaging units of a camera 801 and a camera 802 andacquires images captured from the camera 801 and the camera 802 throughan image acquiring unit 803 and an image acquiring unit 804,respectively. In a pedestrian crossing detecting unit 808, a pedestriancrossing is detected based on The image acquired by the image acquiringunit 803. In a distance calculating unit 805, a distance in a real spaceis calculated for each preset fixed block size using the images acquiredby the image acquiring unit 803 and the image acquiring unit 804. In athree-dimensional object extracting unit 806, the three-dimensionalobject is extracted using the distance obtained by the distancecalculating unit 805. At this time, the extraction of thethree-dimensional object is promoted by adding pedestrian crossingdetection information obtained by a pedestrian crossing detecting unit808. In a three-dimensional object tracking unit 807, three-dimensionalobject information obtained by the three-dimensional object extractingunit 806 is tracked in a chronological order, a relative speed iscalculated, and the results of filtering various parameters of thethree-dimensional object are output. In a control target determiningunit 809, using the information regarding whether or not the pedestriancrossing is present near the three-dimensional object obtained by thepedestrian crossing detecting unit 808, whether or not a trackedthree-dimensional object obtained by the three-dimensional objecttracking unit 807 is a candidate for a control target is determined. Ina control target selecting unit 810, a control target is selected amongcontrol target candidates obtained by the control target determiningunit 809, and information of the selected control target is transmittedto a vehicle control device 811.

Hereinafter, the configuration of each unit shown in FIG. 8 will bedescribed in detail.

The camera 801 and the camera 802 correspond to the camera 101 and thecamera 102 of the stereoscopic camera apparatus 100 according toExample 1. The image acquiring unit 803 and the image acquiring unit 804correspond to the image acquiring unit 103 and the image acquiring unit104 of the stereoscopic camera apparatus 100 according to Example 1. Thedistance calculating unit 805 corresponds to the distance calculatingunit 105 of the stereoscopic camera apparatus 100 according toExample 1. The three-dimensional object tracking unit 807 corresponds tothe three-dimensional object tracking unit 107 of the stereoscopiccamera apparatus 100 according to Example 1. The control targetselecting unit 810 corresponds to the control target selecting unit 110of the stereoscopic camera apparatus 100 according to Example 1. Thevehicle control device 811 corresponds to the vehicle control device 111of the stereoscopic camera apparatus 100 according to Example 1.Accordingly, the above units will not be described to avoid therepetition of the description.

The pedestrian crossing detecting unit 808 determines whether or not thepedestrian crossing is present in the image using the image obtained bythe image acquiring unit 103. Specifically, first, as shown in FIG. 9, aleft pedestrian crossing detection frame 901 and a right pedestriancrossing detection frame 902 are set on the image taken into the imageacquiring unit 803. Regarding the installation positions of thepedestrian crossing detection frames 901 and 902, upper ends thereof arepositioned substantially in the middle of the image, lower ends arepositioned to match lower ends of the image, and left and right endsthereof are positioned to match left and right ends of the image,respectively. In addition, the width of the pedestrian crossingdetection frames is set as, for example, 3 [pix]. The left pedestriancrossing detection frame 901 is used during a left turn, and the rightpedestrian crossing detection frame 902 is used during a right turn.

Next, as in the case of Example 1, a luminance projection graph 602 iscreated by using each of the pedestrian crossing detection frames 901and 902 as one of the strips shown in FIG. 6. As a result, a luminancedistribution graph 603 is created. By obtaining a distribution of theluminance distribution graph 603, at this time, the pedestrian crossinglikelihood can be obtained in the pedestrian crossing detection frames901 and 902 of the image. When the pedestrian crossing likelihood istracked in a chronological order, during a left turn, the pedestriancrossing starts to appear from the right side (A point side) thereof,and an occupancy ratio of the shape of the pedestrian crossing in thepedestrian crossing detection frames gradually increases. After theratio reaches a peak, the ratio gradually decreases. This phenomenon isexpressed by the variance of the luminance distribution graph 603.First, the variance is low, and when the image of the pedestriancrossing starts to appear in the pedestrian crossing detection frames,the variance gradually increases. After the variance reaches a peak, thevariance gradually decreases. At this time, when the three-dimensionalobject is positioned on the pedestrian crossing, the three-dimensionalobject covers the shape of the pedestrian crossing, and thus thevariance decreases. Using this characteristic, when the variancegradually increases to be a set threshold value or more, it isdetermined that a vehicle turns left (or right) on the pedestriancrossing. Next, when the variance rapidly decreases by a set thresholdvalue or more, it is determined that the three-dimensional object may beappeared from a left end (during a right turn, a right end) of theimage, and a left covering flag (during a right turn, a right coveringflag) is set.

With this method, a pedestrian who starts to cross the pedestriancrossing cannot be accurately detected because the pedestrian covers asmall area of the pedestrian crossing. However, in this state, thedistance between the vehicle and the pedestrian is long. Therefore, itis determined that the risk is low, and it is considered that there isno problem.

A process using the three-dimensional object extracting unit 806 issubstantially the same as the process using the three-dimensional objectextracting unit 106 in Example 1. However, in order to detect thethree-dimensional object in an early stage, when the left covering flagobtained by the pedestrian crossing detecting unit 808 (during a rightturn, the right covering flag) is set and the grouped three-dimensionalobject is present near the pedestrian crossing detection frame on theleft end of the image (during a right turn, the right end), thethreshold value of the width (variable depending on the distance) isdecreased to promote the extraction of the three-dimensional object.

The control target determining unit 809 determines a candidate for acontrol target using the tracked three-dimensional object obtained bythe three-dimensional object tracking unit 807 and the covering flagsobtained by The pedestrian crossing detecting unit 808. FIG. 10 shows aprocess flow during a left turn (during a right turn, left is replacedwith right). When the left covering flag is set on each trackedthree-dimensional object, and when the tracked three-dimensional objectis positioned in a left end region, a simple reliability verifyingprocess 1001 is performed. Otherwise, a normal reliability verifyingprocess 1002 is performed. Here, as in the case of the simplereliability verifying process 701 of Example 1, the simple reliabilityverifying process 1001 is performed by reducing verification conditionsto be more gentle than that of the normal reliability verifying process1002 or without verification. “In the left end region” described aboverefers to a region having a width in which the pedestrian appears fromthe left end of the image. This width is appropriately changed inconsideration of the distance to the pedestrian. As a result, the safetyagainst the abrupt appearance of the pedestrian can be improved.

In the stereoscopic camera apparatus according to Example 2, thepedestrian crossing detecting process is performed before thethree-dimensional object extracting process. Therefore, thethree-dimensional object extracting process can be performed using theresult of the pedestrian crossing detecting process, and thethree-dimensional object extracting process can be simplified.Accordingly, the process speed of the entire apparatus can be improved.

In Examples 1 and 2, the case where the pedestrian covers the pedestriancrossing when a vehicle turns right or left has been described as anexample. However, in the stereoscopic camera apparatus according to thepresent invention, the detection can be promoted on a pedestriancrossing which appears on the front of a vehicle when the vehicletravels straight. In this case, for example, an image near thethree-dimensional object is scanned in a horizontal direction from aposition slightly below the lower end of the three-dimensional object,and a portion where a contrast change is irregular is estimated as thecovering of the pedestrian crossing with the three-dimensional object.

In addition, in Examples 1 and 2, the to-be-covered object is thepedestrian crossing. However, in the stereoscopic camera apparatusaccording to the present invention, the same process can be realizedusing a traffic sign having a known shape other than a pedestriancrossing, for example, using characters such as “STOP”. In this case,using affine transformation, a sign image on a road is transformed intoan image of the road when seen from above. This transformed image iscompared to an image of “STOP” which has been previously learned, andthe sign “STOP” is identified. In addition, a portion having a largedifference is extracted, and when the three-dimensional object ispresent in the portion, the covering with the three-dimensional objectcan be detected. Therefore, as in Examples 1 and 2, the determinationand selection of the control target can be promoted. Further, inExamples 1 and 2, the three-dimensional object is the pedestrian.However, the process can be performed by using another vehicle as thethree-dimensional object.

Further, in the stereoscopic camera apparatus according to the presentinvention, even when a road structure having a high linearity and a highprobability of being the same color such as a pavement line, acurbstone, or a guardrail is set as the to-be-covered object, the sameprocess can be performed. In this case, a white line will be describedas an example. A white line candidate point, which is a portion wherethe contrast is changed in order from dark→bright→dark from a lower endof the image, is calculated as a lower half of the image. By connectingthe white line candidate points as left and right white lines, the whiteline can be detected. When the detection result is significantly shorterthan a distance capable of detecting the white line, and when athree-dimensional object is detected in a portion where the detectioncannot be performed, a process of promoting the same detection as thatof Examples 1 and 2 can be realized by estimating the covering with thethree-dimensional object.

REFERENCE SIGNS LIST

-   100: STEREOSCOPIC CAMERA APPARATUS ACCORDING TO EXAMPLE 1-   101, 102: CAMERA-   103, 104: IMAGE ACQUIRING UNIT-   105: DISTANCE CALCULATING UNIT-   106: THREE-DIMENSIONAL OBJECT EXTRACTING UNIT-   107: THREE-DIMENSIONAL OBJECT TRACKING UNIT-   108: PEDESTRIAN CROSSING DETECTING UNIT-   109: CONTROL TARGET DETERMINING UNIT-   110: CONTROL TARGET SELECTING UNIT-   111: VEHICLE CONTROL DEVICE-   201, 202, 203: IMAGE-   302, 303: LENS-   304, 305: IMAGING PLANE-   501, 502: PEDESTRIAN CROSSING DETECTION FRAME-   601: DIVIDED STRIPS-   602: LUMINANCE PROJECTION GRAPH-   603: LUMINANCE DISTRIBUTION GRAPH-   701: SIMPLE RELIABILITY VERIFYING PROCESS-   702: NORMAL RELIABILITY VERIFYING PROCESS-   800: STEREOSCOPIC CAMERA APPARATUS ACCORDING TO EXAMPLE 2-   801, 802: CAMERA-   803, 804: IMAGE ACQUIRING UNIT-   805: DISTANCE CALCULATING UNIT-   806: THREE-DIMENSIONAL OBJECT EXTRACTING UNIT-   807: THREE-DIMENSIONAL OBJECT TRACKING UNIT-   808: PEDESTRIAN CROSSING DETECTING UNIT-   809: CONTROL TARGET DETERMINING UNIT-   810: CONTROL TARGET SELECTING UNIT-   811: VEHICLE CONTROL DEVICE-   901: LEFT PEDESTRIAN CROSSING DETECTION FRAME-   902: RIGHT PEDESTRIAN CROSSING DETECTION FRAME-   1001: SIMPLE RELIABILITY VERIFYING PROCESS-   1002: NORMAL RELIABILITY VERIFYING PROCESS

1. A stereoscopic camera apparatus comprising: three-dimensional objectdetecting means for calculating at least a distance to athree-dimensional object in a real space, a horizontal position, and awidth by stereoscopy based on images captured from first and secondimage sensors; to-be-covered object detecting means for detecting ato-be-covered object having a known shape based an image captured fromone of the first and second image sensors; and to-be-covered objectcovering-detecting means for detecting whether or not thethree-dimensional object detected by the three-dimensional objectdetecting means covers the to-be-covered object when the to-be-coveredobject is detected by the to-be-covered object detecting means, whereinwhen it is detected that the three-dimensional object covers theto-be-covered object, the to-be-covered object covering-detecting meansdetects the three-dimensional object which covers the to-be-coveredobject by performing a process for determining whether or not thethree-dimensional object is present in a simpler way, as compared to acase where it is detected that the three-dimensional object does notcover the to-be-covered object.
 2. The stereoscopic camera apparatusaccording to claim 1, wherein a process using the three-dimensionalobject detecting means is performed before a process using theto-be-covered object detecting means.
 3. The stereoscopic cameraapparatus according to claim 2, wherein the three-dimensional objectdetecting means includes distance calculating means for calculating adepth map by stereoscopy based on the images captured from the first andsecond image sensors, three-dimensional object extracting means forobtaining an extracted three-dimensional object having a position, adistance, and a width as parameters based on the depth map obtained bythe distance calculating means, and three-dimensional object trackingmeans for obtaining a tracked three-dimensional object having aposition, a distance, a width, and a relative speed, which are filteredin a chronological order, as parameters based on the extractedthree-dimensional object obtained by the three-dimensional objectextracting means, and the to-be-covered object covering-detecting meanssets pedestrian crossing detection frames for determining whether or notthe pedestrian crossing is present on left and right sides of positioninformation of the tracked three-dimensional object obtained by thethree-dimensional object tracking means.
 4. The stereoscopic cameraapparatus according to claim 1, wherein a process using theto-be-covered object detecting means is performed before a process usingthe three-dimensional object detecting means.
 5. The stereoscopic cameraapparatus according to claim 4, wherein the to-be-covered objectdetecting means sets pedestrian crossing detection frames fordetermining whether or not a pedestrian crossing is present at left andright ends of an image captured from one of the first and second imagesensors.
 6. The stereoscopic camera apparatus according to claim 4,wherein the three-dimensional object detecting means includes distancecalculating means for calculating a depth map by stereoscopy based onthe images captured from the first and second image sensors,three-dimensional object extracting means for obtaining an extractedthree-dimensional object having a position, a distance, and a width asparameters based on the depth map obtained by the distance calculatingmeans, and three-dimensional object tracking means for obtaining atracked three-dimensional object having a position, a distance, a width,and a relative speed, which are filtered in a chronological order, asparameters based on the extracted three-dimensional object obtained bythe three-dimensional object extracting means, and the three-dimensionalobject extracting means extracts the extracted three-dimensional objectbased on the depth map obtained by the distance calculating means andthe to-be-covered object having a known shape obtained by theto-be-covered object detecting means.
 7. The stereoscopic cameraapparatus according to claim 3, wherein when it is detected that thetracked three-dimensional object does not cover the to-be-coveredobject, the to-be-covered object covering-detecting means verifieswhether or not the extracted three-dimensional object matches the sametracked three-dimensional object through a preset number or more ofplural frames such that only the tracked three-dimensional objectsatisfying a condition is registered as a control candidatethree-dimensional object, and when it is detected that the trackedthree-dimensional object covers the to-be-covered object, theto-be-covered object covering-detecting means verifies whether or notthe extracted three-dimensional object matches the same trackedthree-dimensional object through a smaller number of plural frames thanthe preset number of the plural frames such that only the trackedthree-dimensional object satisfying a condition is registered as acontrol candidate three-dimensional object, or the trackedthree-dimensional object is registered as a control candidatethree-dimensional object without verification.
 8. The stereoscopiccamera apparatus according to claim 1, wherein the to-be-covered objectis one or plural three-dimensional objects among a traffic sign paintedon a road, a pavement line including a white line painted on a road, apedestrian crossing painted on a road, and a highly linear roadstructure including a curbstone installed along a road.